Today's computer networks are extremely complex, with hundreds or more of applications, thousands or more of servers, hundreds or more of locations, hundreds of thousands of clients, and network traffic routed by numerous switches and routers on the computer networks. Network and application data collected from various parts of the network can provide insight into network conditions, but the enormous amount of data present a challenge for data storage, processing, and retrieval.
Many conventional network monitoring systems store network data on a first-in-first-out (FIFO) basis. In these network monitoring systems, older data packets are overwritten with the new data packets. When an event is detected (e.g., detection of an unauthorized access to resources), the data packets currently stored in the network monitoring system may be retrieved and analyzed. However, the network monitoring systems have limited storage capacity. Hence, the network monitoring systems store data packets that go back in time for only a limited period of time. Data packets received before this period is deleted from the network monitoring systems, and thus, are unavailable for troubleshooting or analysis.
Storing the data packets for a longer period of time is advantageous because some network analysis may require data packets that were communicated in the network quite some time ago. However, in order to store data packets for a longer period of time, storage devices with larger capacity are needed in the network monitoring system. Moreover, as the number of stored data packets increases, the time needed to store and retrieve the data packets increases accordingly. The increased time for storage and retrieve may hamper prompt diagnosis and taking of remedial actions.